A wide area network (WAN) can connect a local area network (LAN) established by an Internet service provider (ISP) to a data center outside of the LAN. A WAN may employ edge point of presence (PoP) cache systems and content delivery networks (CDNs) to reduce the likelihood of network traffic causing bottlenecks at the data centers. Cache systems can store popular content, e.g., a video or photo that is requested by users in locations that are closer to the users than the data centers, in terms of geographic distance, network distance (“hops”), or both. A “cache hit” occurs when requested data can be found in a cache, whereas a “cache miss” occurs when it cannot. Having a high cache “hit rate” lowers the latency of delivering the content to the user and also reduces the bandwidth requirement because the requested data has to travel a smaller distance than from the data center.
The need for a high cache hit rate suggests the need for a reduced number of cache systems such that each cache system can serve a higher number of requesters. Popular content can be repeatedly served to clients without fetching data from an origin data center. However, in certain applications, such as live broadcasting, a cache system often receives client requests simultaneously or within a short time interval as to be substantially simultaneous. The substantially simultaneous arrival of a large batch of client requests (e.g., greater than two hundred thousand or a million client requests) can increase the likelihood of overloading the cache system. The balance between optimizing for higher hit rate and reducing the potential threat of overloading the cache system remains a challenge in the industry.
The figures depict various embodiments of this disclosure for purposes of illustration only. One skilled in the art will readily recognize from the following discussion that alternative embodiments of the structures and methods illustrated herein may be employed without departing from the principles of embodiments described herein.